The Eigenoption-Critic Framework

Eigenoptions (EOs) have been recently introduced as a promising idea for generating a diverse set of options through the graph Laplacian, having been shown to allow efficient exploration Machado et al. [2017a]. Despite its first initial promising results, a couple of issues in current algorithms limit its application, namely: 1) EO methods require two separate steps (eigenoption discovery and reward maximization) to learn a control policy, which can incur a significant amount of storage and computation; 2) EOs are only defined for problems with discrete state-spaces and; 3) it is not easy to take the environment’s reward function into consideration when discovering EOs. In this paper, we introduce an algorithm termed eigenoptioncritic (EOC) that addresses these issues. It is based on the Option-critic (OC) architecture Bacon et al. [2017], a general hierarchical reinforcement learning algorithm that allows learning the intra-option policies simultaneously with the policy over options. We also propose a generalization of EOC to problems with continuous state-spaces through the Nyström approximation. EOC can also be seen as extending OC to nonstationary settings, where the discovered options are not tailored for a single task.